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Abstract 


We  provide  an  integrative  solution  for  negotiations  in  the  Navy  detailing  process  con¬ 
sidering  the  uncertain  and  dynamic  outside  options.  The  outside  options  influence 
the  negotiation  strategies  via  the  impact  on  the  reservation  prices.  The  solution  is 
composed  of  three  modular  models:  single-threaded  negotiations,  synchronized  multi¬ 
threaded  negotiations,  and  dynamic  multi-threaded  negotiations.  The  single-threaded 
negotiation  model  provides  the  negotiation  strategy  given  the  reservation  price.  The 
other  two  models  calculate  the  reservation  price  of  a  negotiation  thread  based  on  the 
model  of  the  outside  options  by  viewing  all  outside  options  as  a  multi-threaded  nego¬ 
tiation  process.  The  model  of  synchronized  multi-threaded  negotiations  considers  the 
presence  of  concurrently  available  outside  options  and  provides  an  approach  to  esti¬ 
mate  the  outcome  when  the  threads  are  known.  The  model  of  dynamic  multi-threaded 
negotiations  expands  the  synchronized  model  by  considering  the  uncertain  outside  op¬ 
tions  that  may  come  dynamically  in  the  future.  We  propose  two  effective  negotiation 
strategies,  the  time-dependent  strategy  and  Bayesian  learning  strategy,  from  the  AI 
field.  Four  heuristic  approaches  are  designed  to  estimate  the  expected  utility  from 
a  synchronized  multi-threaded  negotiation.  A  Poisson  process  is  used  to  model  the 
random  sequential  arrivals,  and  formulas  are  provided  to  calculate  the  expected  utility 
from  a  negotiation  process  when  uncertain  outside  options  may  come  in  the  future. 

We  give  preliminary  experimental  analysis  to  characterize  the  impact  of  outside  options 
on  the  reservation  price  and  so  on  the  negotiation  strategy.  The  results  show  that  the 
utility  of  a  negotiator  improves  significantly  when  she  considers  outside  options  from 
when  she  does  not,  and  the  average  utility  is  higher  when  she  both  considers  the 
concurrent  outside  options  and  foresee  the  future  ones  than  when  she  only  considers 
the  concurrent  outside  options. 


1  Introduction 


Bilateral  negotiations  is  an  important  mechanism  to  implement  flexible  and  distributed 
matching  in  the  Navy  detailing  system  [4,  7].  In  the  Navy  detailing  process  a  negotiator 
can  face  more  than  one  potential  matching  alternative.  For  example,  a  Command  may 
find  more  than  one  Sailor  that  is  qualified  for  the  job,  and  a  Sailor  can  be  informed  of 
more  than  one  job  vacancies  that  interest  him.  These  alternatives  are  called  outside 
options.  Accepting  a  proposal  in  one  negotiation  means  refusing  all  outside  options. 
On  the  other  hand  one  may  leave  a  negotiation  (called  “opt-out”  of  a  negotiation) 
without  reaching  an  agreement  based  on  the  expectation  of  reaching  a  more  favorable 
agreement  in  outside  options.  Modelling  the  outside  options  and  understanding  the 
interaction  between  outside  options  and  a  negotiation  process  is  an  essential  aspect  to 
designing  an  effective  negotiation  strategy  in  the  Navy  detailing  process. 

Outside  options  can  exist  concurrently  with  a  negotiation,  or  come  sequentially  in  the 
future  [6].  A  concurrently  available  outside  option  is  a  negotiation  thread  that  the 
negotiator  is  involved  in  simultaneously  with  another  thread.  This  happens  because  a 
Command  may  find  multiple  potential  Sailors  that  are  available  for  negotiations  for  the 
same  job  at  the  same  time.  A  Sailor  may  also  be  invited  to  more  than  one  negotiation 
-  one  for  each  potential  job  -  simultaneously.  A  sequentially  available  outside  option  is 
a  matching  opportunity  that  comes  in  the  future.  A  Command  is  not  informed  at  one 
time  of  all  potential  Sailors  that  will  become  available  during  the  whole  search  period, 
neither  is  a  Sailor  aware  of  all  interesting  job  vacancies  during  their  application  period. 
The  information  on  Sailors  and  jobs  that  are  available  is  published  periodically  and 
sequentially.  To  obtain  information  may  also  induce  searching  cost,  which  prevents 
awareness  of  complete  information  at  one  time.  Outside  options  are  uncertain  in 
terms  of  both  availability  and  quality.  The  availability  of  outside  options  is  uncertain 
because  a  negotiator  is  not  sure  when  an  outside  option  is  available  and  how  many 
are  available.  The  quality  of  outside  options  is  uncertain  because  a  negotiator  cannot 
predict  the  outcome  of  a  negotiation  thread,  or  the  preferences  of  the  other  party  in  a 
negotiation  thread.  How  to  model  the  availability  and  uncertainty  of  outside  options 
is  an  important  consideration  for  modelling. 

Outside  options  affect  the  negotiation  strategies  via  their  impact  on  the  reservation 
price.  The  reservation  price  is  the  worst  agreement  that  a  negotiator  can  accept.  For 
example,  in  a  buyer-seller  negotiation  model,  the  reservation  price  of  the  buyer  is  the 
highest  price  she  is  willing  to  pay  for  the  negotiated  good.  For  the  seller,  the  reservation 
price  is  the  lowest  price  at  which  he  is  willing  to  sell  the  good.  The  price  at  which 
the  seller  is  willing  to  sell  depends  on  the  production  cost  of  the  seller.  The  price  at 
which  the  buyer  is  willing  to  buy  depends  on  the  valuation  of  the  buyer  to  the  good. 
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Additionally  they  both  depend  on  the  availability  of  other  buyers  or  sellers.  Let’s  take 
the  standpoint  of  the  buyer.  Similar  statements  can  be  drawn  for  the  seller.  For  the 
buyer,  if  there  are  no  outside  sellers,  the  reservation  price  is  equal  to  the  valuation  . 
However,  the  negotiation  does  not  necessarily  end  up  with  the  reservation  price  because 
the  seller  does  not  know  the  buyer’s  reservation  price.  If  there  are  other  sellers  joining 
in  the  market  and  the  original  seller  is  not  a  monopoly  any  more,  then  the  buyer  will 
decrease  her  reservation  price  hoping  that  she  could  reach  a  deal  with  other  sellers, 
if  she  cannot  reach  an  agreement  with  the  current  seller  with  a  price  less  than  the 
reservation  price.  The  reservation  price  will  be  lower  if  the  buyer  expects  that  there 
are  more  outside  sellers  with  possibly  lower  prices.  The  reservation  price  impacts  the 
proposal  and  response  decisions  of  a  negotiator.  When  there  are  outside  options,  design 
of  an  effective  negotiation  strategy  can  be  divided  into  two  parts:  the  first  is  to  design 
a  negotiation  strategy  given  the  reservation  price  and  other  inputs,  the  second  is  to 
calculate  the  reservation  price  based  on  the  model  of  outside  options. 

In  our  previous  modelling  work  [6]  we  have  proposed  a  nested  model  for  negotiations 
in  the  Navy  detailing  process  considering  the  uncertain  and  dynamic  outside  options. 
The  model  is  composed  of  three  modules,  single-threaded  negotiations,  synchronized 
multi-threaded  negotiations,  and  dynamic  multi-threaded  negotiations.  These  three 
models  embody  increased  sophistication  and  complexity.  The  single-threaded  negoti¬ 
ation  model  provides  the  negotiation  strategies  without  specifically  considering  out¬ 
side  options.  The  model  of  synchronized  multi-threaded  negotiations  builds  on  the 
single-threaded  negotiation  model,  and  considers  the  presence  of  concurrently  avail¬ 
able  outside  options  by  calculating  the  reservation  price  based  on  the  other  existing 
negotiation  threads.  The  model  of  dynamic  multi-threaded  negotiations  expands  the 
synchronized  multi-threaded  model  by  considering  the  uncertain  outside  options  that 
may  come  dynamically  in  the  future. 

In  this  report  we  present  specific  solutions  of  these  models.  We  propose  two  effective 
negotiation  strategies,  the  time-dependent  strategy  and  Bayesian  learning  strategy, 
from  the  A1  field.  Four  heuristic  approaches  are  designed  to  estimate  the  expected 
utility  from  a  synchronized  multi-threaded  negotiation.  A  Poisson  process  is  used 
to  model  the  random  sequential  arrivals,  and  formulas  are  provided  to  calculate  the 
expected  utility  from  a  negotiation  process  when  uncertain  outside  options  may  come  in 
the  future.  Empirical  analysis  is  provided  to  characterize  the  impact  of  outside  options 
on  the  reservation  price  and  therefore  on  the  negotiation  strategy.  The  results  show 
that  the  utility  of  a  negotiator  improves  significantly  when  she  considers  outside  options 
from  when  she  does  not,  and  the  average  utility  is  higher  when  she  both  considers  the 
concurrent  outside  options  and  foresees  the  future  ones  than  when  she  only  considers 
the  concurrent  outside  options. 
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The  rest  of  the  report  is  organized  as  follows:  Section  2  presents  the  models  and 
solutions.  In  Section  3  we  describe  the  experimental  results.  Section  4  concludes. 


2  The  model 

For  the  convenience  of  presentation  we  call  the  two  agents  in  a  bilateral  negotiation  a 
buyer  and  a  seller,  and  we  present  the  model  from  the  buyer’s  perspective.  The  buyer 
prefers  a  lower  value  of  the  negotiated  issue  while  the  seller  prefers  a  higher  value.  In 
the  Navy  detailing  process  we  can  regard  a  Command  as  a  buyer  and  a  Sailor  as  a 
seller.  The  roles  of  a  buyer  and  a  seller  are  interchangeable  by  changing  the  sign  of  the 
value  of  the  negotiated  issue. 

There  are  T  periods  over  the  entire  horizon  of  a  detailing  window.  Let  a  period  be 
denoted  by  t,  t  —  0, . . . ,  T—  1.  A  buyer  needs  to  reach  an  agreement  with  a  seller  before 
period  T.  The  potential  sellers  may  come  in  different  time  unexpectedly  with  different 
reservation  prices,  and  the  buyer  can  negotiate  with  the  sellers  simultaneously.  The 
negotiation  between  the  buyer  and  a  seller  is  called  a  negotiation  thread.  The  number 
of  threads  in  period  t  is  denoted  by  nt,  and  the  collection  of  threads  in  period  t  is 
denoted  by  Dt  =  {dj}”L1.  The  seller  in  the  thread  d*  is  denoted  by  s*.  Based  on 
the  background  information  on  the  sellers  (or  sellers’  products),  the  buyer  can  value 
the  sellers  (or  the  sellers’  products)  differently.  For  example,  a  Command  can  attach 
different  values  to  having  the  job  filled  by  different  Sailors  based  on  the  Sailors’  skills 
and  experiences  and  the  job’s  requisites.  A  Sailor  can  also  have  different  preferences 
on  different  jobs  because  of  the  difference  in  the  location  or  properties.  Let  the  value 
of  the  seller  s;  be  V{.  If  the  buyer  reaches  an  agreement  with  the  seller  Si  at  x ,  then 
the  utility  of  the  buyer  is  Vi  —  x. 

The  buyer  wants  to  reach  the  lowest  possible  agreement  with  a  seller.  But  she  does 
not  know  the  bottom  line  of  the  seller,  or  what  price  is  acceptable  to  the  seller.  If  the 
price  she  insists  on  is  low,  she  could  get  a  high  profit  but  risk  losing  the  cooperation 
opportunity  with  the  seller.  On  the  other  hand  if  the  price  she  agrees  on  is  high, 
she  can  make  a  deal  with  the  seller  with  high  probability  but  then  the  profit  is  low. 
Although  the  buyer  does  not  know  the  reservation  price  of  a  seller,  she  can  have  some 
estimation  of  the  information  based  on  statistical  aggregation  of  the  historical  data  or 
survey  work.  The  historical  data  records  the  agreements  that  were  reached  on  the  same 
or  similar  products  (jobs)  in  the  past.  A  negotiator  can  also,  maybe  by  the  help  of  a 
third  party,  do  a  survey  to  ask  the  reservation  prices  of  a  representative  population. 
The  estimation  of  the  reservation  price  of  a  seller  is  characterized  by  a  probability 
distribution  F(-),  where  F(x)  denotes  the  probability  that  the  reservation  price  of  a 
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seller  is  no  greater  than  x.  This  probability  distribution  is  called  the  prior  belief  of  the 
buyer.  A  negotiation  provides  a  mechanism  for  the  negotiators  to  exchange  messages 
and  adjust  their  proposals.  Usually  a  negotiator  will  start  with  a  favorable  proposal, 
and  then  make  sequential  concession  until  a  proposal  is  accepted  or  the  negotiation 
deadline  is  reached.  The  negotiation  strategy  decides  the  pace  of  concession  at  each 
step  based  on  the  single-threaded  negotiation  model  given  the  buyer’s  reservation  price 
and  the  prior  belief. 

When  there  are  outside  options  the  decision  of  a  negotiator  is  more  complicated.  The 
buyer  will  expect  to  reach  a  utility  that  is  no  worse  than  the  expected  utility  that 
she  could  achieve  from  the  outside  options.  In  other  words,  the  buyer  has  a  threshold 
on  the  lowest  utility  that  she  should  achieve  from  a  negotiation  thread  based  on  the 
expectation  of  the  outside  options.  The  lowest  utility  to  achieve  in  thread  di  is  called  the 
reservation  utility  OUi  in  the  thread.  The  reservation  utility  is  equal  to  the  expected 
utility  from  the  outside  options,  which  can  be  viewed  together  as  a  multi-threaded 
negotiation.  Given  the  reservation  utility  OUi ,  the  reservation  price  Ri  of  the  buyer 
in  thread  di  can  be  calculated  by  Ri  =  Vi  —  OUi.  If  there  are  no  outside  options,  we 
can  say  that  the  reservation  utility  is  zero,  or  the  reservation  price  Ri  is  equal  to  the 
value  Vi .  Because  of  the  heterogeneity  among  the  sellers,  the  reservation  prices  in  each 
thread  may  be  different.  If  the  reservation  price  in  each  thread  is  known,  the  buyer 
can  apply  the  single-threaded  negotiation  model  to  make  the  negotiation  decisions  in 
each  thread. 

Calculation  of  the  expected  utility  from  the  outside  options  depends  on  the  model  on 
the  outside  options,  and  on  the  approach  to  estimate  the  expected  utility  from  a  multi¬ 
threaded  negotiation.  In  a  synchronized  multi-threaded  negotiation  model  the  outside 
options  at  period  t  for  thread  di  are  the  other  concurrently  existing  negotiation  threads 
Dt  \  dt.  The  synchronized  model  maps  the  current  outside  options  to  the  reservation 
utility  OUi(Dt  \  di)  of  each  thread  di,  i  =  l,...,rq.  The  dynamic  multi-threaded 
negotiation  model  also  considers  the  outside  options  that  may  come  in  the  future  at 
uncertain  times  with  uncertain  values.  Let  the  probability  that  a  new  opponent  arrives 
in  a  period  be  p,  and  the  probability  distribution  of  the  value  of  an  opponent  be  <!>(•), 
where  $(v)  is  the  probability  that  the  value  of  an  opponent  is  less  than  v.  The  dynamic 
multi-threaded  negotiation  model  calculates  the  reservation  utility  OUi(Dt  \  df\p ,  $(•)) 
for  each  current  thread  di  based  on  the  current  outside  options  Dt\di ,  given  the  arrival 
probability  and  the  probability  distribution  of  opponents’  values.  The  dynamic  multi¬ 
threaded  negotiation  model  can  be  viewed  as  a  synchronized  model  with  uncertain 
threads. 

In  Section  2.1  we  first  present  the  negotiation  strategy  solution  in  a  single-threaded 
negotiation.  In  Section  2.2  the  influence  of  the  concurrent  negotiation  threads  is  quan- 
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tified  in  the  reservation  utility.  In  Section  2.3  the  negotiation  threads  that  may  come 
sequentially  in  the  future  are  considered  additionally  and  the  impact  is  integrated  in 
the  reservation  utility  and  thus  in  the  negotiation  strategy. 

2.1  Single-threaded  negotiations 

We  describe  the  negotiations  based  on  an  alternating-offers  negotiation  protocol,  be¬ 
cause  (1)  it  is  a  sequential  negotiation  protocol,  which  allows  negotiators  to  dynamically 
adjust  the  offers  and  does  not  require  reasoning  and  computation  as  complicated  as  in 
a  one-shot  negotiation;  and  (2)  it  provides  more  flexibility  for  the  negotiating  parties 
to  efficiently  convey  information  than  an  ultimatum  negotiation  protocol,  in  which  one 
party  proposes  and  the  other  party  can  only  respond  by  accepting  or  rejecting  the 
offers  [1,  12].  The  negotiation  strategy  based  on  an  alternating-offers  protocol  specifies 
the  decisions  for  both  proposal  generation  and  response  to  an  offer. 

In  a  negotiation  following  an  alternating-offers  protocol,  the  negotiators  propose  and 
respond  alternatively,  until  one  accepts  an  offer  or  quits  the  negotiation1,  or  the  nego¬ 
tiation  deadline  T  is  reached.  The  history  H*  of  a  negotiation  at  time  t,  t  >  1,  is  a 
sequence  of  the  negotiators’  actions  before  t,  i.e. ,  H 1  =  A™m<v  where  A™  is  the  action 
of  negotiator  i  at  time  m.  Therefore  the  history  of  an  alternating-offers  negotiation 
at  time  t  is  a  sequence  of  proposals,  i.e.,  Ht  =  {xla,xl,x^a,xl, . . .  ^^(x^)},  where  x ™ 
is  the  proposal  submitted  by  negotiator  i  at  time  m.  Generally  a  negotiation  strategy 
Si  specifies  the  action  at  each  step  conditional  on  the  negotiation  history,  and  based 
on  the  reservation  price  and  prior  belief,  i.e.,  A\  =  Si(Ht\Ri}  Fi(-)),  0  <  t  <  T,  where 
A\  e  {accept,  reject  and  propose  xl+1,quit}.  To  give  an  optimal  negotiation  strat¬ 
egy,  game  theoretic  analysis  is  required  to  derive  the  perfect  Bayesian  equilibrium2  [3] . 
The  analysis  of  the  perfect  Bayesian  equilibrium  is  not  tractable  when  both  parties 
have  incomplete  information  with  an  alternating-offers  protocol,  although  there  have 
been  conclusions  on  the  optimal  strategy  in  bilateral  negotiations  with  two-sided  in¬ 
complete  information  in  a  direct  revelation  mechanism  when  the  prior  beliefs  of  both 
parties  follow  a  uniform  distribution  [11]  [3,  Chapter  7].  We  adopt  two  effective  negoti¬ 
ation  strategies  that  have  been  developed  in  the  AI  held  and  proved  successful.  These 
two  strategies  are  the  time- dependent  strategy  [2]  and  Bayesian-learning  strategy  [17]. 
These  approaches  do  not  explicitly  model  the  strategic  interactions  between  the  ne- 

1A  negotiator  could  quit  the  negotiation  because  an  agreement  is  reached  in  another  negotiation 
thread 

2The  strategies  of  players  constitute  a  perfect  Bayesian  equilibrium,  if  given  the  strategies  of  the 
other  players,  a  player  cannot  obtain  strictly  better  profit  on  expectation  in  each  subgame  by  deviating 
to  another  sequential  strategy. 


5 


gotiators.  Instead  they  focus  on  some  issues  that  are  important  to  the  decision  and 
information,  and  provide  flexible  heuristic  decision  functions. 

2.1.1  Time-dependent  approach 

The  time-dependent  approach  focuses  on  the  impact  of  time  on  negotiations.  A  nego¬ 
tiator  usually  has  a  hard  time  deadline  before  which  the  negotiation  has  to  end.  In  the 
Navy  detailing  system  a  Sailor  has  a  certain  detailing  time  window  which  is  usually 
three  months.  After  that,  if  the  Sailor  has  not  located  a  job  he  has  to  be  allocated 
by  a  detailcr.  Similarly  a  Command  has  to  find  a  Sailor  to  fill  in  a  job  before  certain 
time.  With  less  remaining  negotiation  time,  a  negotiator  is  more  pressed  to  reach  an 
agreement  and  to  concede.  But  a  negotiator  cannot  wait  until  the  last  moment  to 
concede  because  the  time  is  also  valuable.  The  time  spent  on  negotiation  should  be 
reasonable  with  respect  to  the  value  of  the  agreement  that  is  reached.  A  negotiator 
who  persistently  holds  to  the  price  risks  losing  the  negotiation  opponent  because  the 
opponent  may  find  another  partner  during  the  process.  There  can  be  other  factors 
that  also  impact  the  negotiation  strategy  such  as  the  available  resources.  Actually  the 
remaining  time  can  be  viewed  as  one  type  of  resource.  The  same  approach  can  be  used 
to  model  the  impact  of  other  resources,  if  there  are  any,  and  the  decision  can  add  to 
the  decision  based  on  the  time-dependent  strategy. 

In  the  time-dependent  approach  time  is  the  predominant  factor  used  to  decide  which 
proposal  to  offer  or  accept  next.  For  the  buyer  in  a  negotiation  thread  di,  the  proposal 
to  offer  or  accept  is  within  the  interval  [■ mini,  max i\ ,  where  maxi  is  the  reservation 
price  of  the  buyer  in  thread  di,  and  mini  is  the  lower  bound  of  a  valid  offer  (we  can 
reasonably  assume  mini— 0)-  For  a  seller  mini  will  be  the  reservation  price  and  maxi  is 
the  upper  bound  of  a  valid  offer.  Initially  a  negotiator  offers  the  most  favorable  value 
for  herself.  If  the  proposal  is  not  accepted,  a  negotiator  concedes  with  time  proceeding 
and  moves  toward  the  other  end  of  the  interval.  The  pace  of  concession  depends  on  the 
negotiation  strategy  and  is  characterized  by  a  function  «*(£)  of  time.  The  value  x\  to 
be  offered  by  a  buyer  and  the  value  x*s  to  be  offered  by  a  seller  at  time  t,  t  G  [0,  T  —  1], 
are  as  follows: 

x\  =  mina  +  aa(t)(maxa  —  mina )  (1) 

xl  =  min\,  +  (1  —  ab(t))(maxb  —  miub).  (2) 

The  buyer  accepts  an  offer  xl  from  negotiator  b  at  time  t  if  it  is  not  worse  than  the 
offer  he  would  submit  next  time,  i.e.,  x *+1  >  x\.  Similarly  the  negotiator  b  accepts  an 
offer  xl  at  time  t  if  x\+1  <  xl . 
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The  time-dependent  function  can  be  defined  by  a  family  of  polynomial  functions3: 

«*(*)  =  (f)h  (3) 

The  constant  /3  >  0  determines  the  concession  pace  along  with  time,  or  the  convexity 
degree  of  the  curve  of  proposals  (see  Figure  1).  By  varying  (3  a  wide  range  of  negotiation 
strategies  can  be  characterized.  Two  sets  of  (3  can  be  identified  to  characterize  two 
classes  of  strategies:  Boulware  with  (3  <  1  and  Conceder  with  (3  >  1.  With  a  Boulware 
strategy  a  negotiator  tends  to  maintain  the  offered  value  until  the  time  is  almost 
exhausted,  then  she  concedes  to  the  reservation  price  quickly.  With  a  Conceder  strategy 
a  negotiator  goes  to  the  reservation  price  rapidly  and  early.  Figure  1  shows  the  change 
of  offers  with  time  in  the  two  strategy  classes.  Which  strategy  to  use  depends  on  how 
much  a  negotiator  values  the  time  and  the  expectation  of  the  opponent’s  strategy. 
An  impatient  negotiator  wants  to  reach  a  deal  earlier  and  is  more  likely  to  follow  the 
Conceder  strategy.  If  a  negotiator  expects  the  opponent  to  be  a  conceder,  she  will  tend 
to  apply  a  Boulware  strategy. 

The  time-dependent  strategy  is  intuitive  and  simple,  and  has  been  proved  useful  in 
real  applications  [2],  The  shape  of  the  curve  of  concession,  or  the  parameter  f3,  is  what 
differentiates  the  strategies  of  a  negotiator.  The  disadvantage  of  the  approach  is  that 
the  real-time  information  in  the  negotiation  is  not  used.  Once  (3  is  chosen,  the  offer 
curve  is  pre-determined.  But  the  bilateral  negotiation  based  on  an  alternating-offers 
protocol  is  a  sequential  interactive  process.  The  information  that  has  been  revealed  by 
the  opponent  in  the  negotiation  can  be  useful  in  making  subsequent  decisions.  This 
consideration  is  included  in  the  Bayesian-learning  strategy. 

2.1.2  Bayesian-learning  strategy 

In  the  Bayesian  learning  approach  a  negotiation  agent  uses  the  Bayesian  framework  to 
update  her  prior  knowledge  and  belief  about  the  environment  and  other  agents  based 
on  the  messages  that  have  been  exchanged  previously  in  the  negotiation  and  domain 
knowledge.  Based  on  her  increasingly  accurate  belief,  the  negotiator  can  make  better 
sequential  decisions  in  the  negotiation. 

3  Alternatively  we  can  also  use  the  exponential  function  family,  and  define 

ai(t)  =  e(1”T)'3. 

These  two  families  are  similar  in  their  functionality  except  that  their  sensitivity  to  the  change  of  time 
is  different  with  different  (3-  For  the  same  big  value  of  /?,  the  polynomial  function  concedes  faster 
at  the  beginning  than  the  exponential  one;  then  they  behave  similarly.  For  a  small  value  of  (3 ,  the 
exponential  function  waits  longer  than  the  polynomial  one  before  it  starts  conceding  [2]. 
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Figure  1:  Offer  curves  with  different  /3 


Let  the  possible  type  of  the  opponent  be  in  the  collection  {hj}™=1.  As  we  have  defined 
in  [6] ,  the  type  of  a  negotiator  is  the  private  information  held  by  the  agent  that  impacts 
the  negotiation  outcome.  The  reservation  price  is  the  type  of  an  opponent.  The  prior 
belief  is  the  probability  that  the  opponent  has  a  type  hj  and  is  denoted  by  P(hj), 
j  =  1 , ...  ,n.  The  domain  knowledge  attaches  a  probability  P(e\hj)  to  every  possible 
action  e  of  the  opponent  conditional  on  the  type  hj.  An  opponent  proposal  action  can 
be  viewed  as  a  signal  of  the  opponent’s  type.  Given  the  encoded  domain  knowledge 
in  the  form  of  conditional  probabilities  and  the  signal  e  given  by  the  opponent,  a 
negotiator  can  nse  the  standard  Bayesian  updating  rule  to  revise  her  belief  about  the 
opponent’s  type: 

pMFsM 

EL,  P(e\ht)P(hty 

Given  her  revised  belief  the  negotiator  can  apply  its  decision  rule  to  make  a  proposal 
or  respond  to  an  offer.  The  decision  rule  can  be  a  simple  strategy,  for  example,  (for 
the  buyer)  to  propose  a  price  which  is  10%  below  the  estimated  reservation  price 
of  the  seller.  Or  it  can  be  a  solution  to  an  optimization  problem  which  provides 
decision  heuristics  For  example,  to  make  a  proposal  that  maximizes  her  expected  utility 
assuming  the  negotiation  ends  next  period.  Then  the  proposal  of  the  buyer  at  period 
t  is  the  solution  of  argmaxxFt(x){v  —  x),  where  v  is  the  value  of  the  seller,  Ft(x)  is  the 
probability  that  the  seller’s  reservation  price  is  less  than  x  based  on  the  revised  belief 
at  period  t.  The  decision  of  the  buyer  trades  off  between  the  probability  of  the  proposal 
being  rejected  and  the  profit  if  the  proposal  is  accepted.  The  solution  suggests  that 
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the  next  proposal  xt  satisfies 


v  -  xt  = 


(5) 


F(xt) 

F'(xty 


The  domain  knowledge  can  be  very  specific  and  confirmative,  for  example,  “in  our 
business  a  seller  usually  offers  a  price  17%  above  the  reservation  price”.  It  can  also 
be  simple  and  “natural”  (and  cannot  be  called  “domain”  knowledge  anymore),  for 
example,  a  seller  will  not  offer  a  price  that  is  lower  than  her  reservation  price.  While 
specific  domain  knowledge  allows  efficient  update  of  the  prior  belief,  domain  knowledge 
is  hard  to  identify  and  acquire.  It  also  requires  discretization  of  the  type  space  to 
apply  the  Bayesian  framework,  as  is  shown  in  Equation  4.  The  “natural”  knowledge 
does  not  help  in  the  modelling  and  updating  of  the  prior  belief  as  much  as  specific 
domain  knowledge,  but  it  is  easy  to  acquire  and  may  not  need  a  discrete  type  space. 
For  example,  let  Ft(x)  be  the  current  belief  at  period  t  on  the  probability  that  the 
reservation  price  of  a  seller  is  less  than  x,  x  €  [x,  x\ .  If  the  seller  next  proposes  a  price 
z  G  [x,x\,  then  the  belief  can  be  revised  to  Ft+ 1(')  based  on  the  knowledge  that  a 
seller’s  reservation  price  is  always  less  than  her  proposal.  Then  the  new  belief  is: 


Ft+i(x) 


for  x  G  [x,  z] 
Ft+i(z)  for  x  G  (z,x ]. 


Both  the  belief  update  method  (6)  and  the  decision  function  (5)  have  been  applied  in 
previous  work  in  negotiations  [8,  15],  and  in  other  problems  such  as  bidding  in  double 
auctions  [16,  5].  They  can  be  used  to  build  a  basic  Bayesian- learning  negotiation 
strategy  if  no  other  domain  knowledge  or  more  efficient  decision  rule  is  available. 


2.2  Synchronized  multi-threaded  negotiations 

In  a  synchronized  multi-threaded  negotiation  process  a  negotiator  participates  in  mul¬ 
tiple  bilateral  negotiation  threads  with  different,  simultaneous  negotiation  opponents. 
The  negotiator  can  reach  an  agreement  in  at  most  one  of  these  threads,  and  is  aware 
of  all  the  threads  at  the  beginning  of  the  process.  From  one  thread’s  perspective  the 
other  threads  are  outside  options.  The  reservation  utility  that  the  negotiator  should 
set  in  one  thread  is  equal  to  the  expected  utility  from  all  other  threads.  The  other 
threads  form  a  synchronized  multi-threaded  negotiation  with  one  less  thread  than  the 
original  process. 

As  we  have  explained  in  [6],  with  a  multi-threaded  negotiation  it  is  reasonable  to  assume 
that  if  any  agreement  is  reached,  the  agreement  is  signed  with  the  most  competitive 
opponents  among  all  opponents  of  the  threads.  For  a  buyer  the  seller  Si  in  thread  di 
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is  more  competitive  than  the  sellers  in  other  threads  if  st  can  give  more  utility  to  the 
buyer,  i.e.,  y*  =  Vi  —  ri  is  greater  than  y3 ,  dj  G  D  \  di:  where  rl  is  the  reservation 
price  the  seller  in  thread  dt ,  and  D  =  {di, . . .  ,d^}  is  the  collection  of  threads.  The 
amount  y*  is  the  maximum  utility  that  the  buyer  can  achieve  from  the  negotiation 
thread  di.  Because  the  buyer  does  not  know  the  reservation  price  of  a  seller,  he  does 
not  know  the  maximum  utility  in  a  thread  either.  Based  on  the  prior  belief  F(-)  on 
the  reservation  price  of  a  seller,  the  negotiator  can  derive  the  probability  distribution 
of  the  maximum  utility.  From  the  probability  distribution  of  the  maximum  utility  in 
every  thread,  the  probability  distribution  of  the  highest  and  second  highest  maximum 
utility  can  be  calculated.  Let  Gi(y)  denote  the  probability  of  the  maximum  utility  in 
thread  di  being  less  than  y.  Let  G1(y)  and  G2(y)  be  the  probability  distribution  of  the 
highest  and  second  highest  maximum  utilities.  These  probabilities  can  be  calculated 
by  the  following  formulas: 

Gi(y)  =  Pr(v i  -  n  <y)  =  Pr(r,  >vt-y)  =  F(v,  -  y) 

G\y)  =  I]  G,(y) 

di&D 

N 

G\y)  =  G\y)  +  £(1  -  G,{y))  n  Gj(y). 

i= 1  dj£D\di 

The  corresponding  probability  density  functions,  or  the  derivatives  of  these  (cumula¬ 
tive)  probability  distribution  functions,  are  as  follows: 

9d(y)  =  - f(vd-y ) 

9\y)  =  J2  9i(y )  II  Gi(y) 

di^D  dj{LD\di 

N  N 

g2(y)  =  g\y)  II  GAv)  +  EC1  -  Gi(y))[  Y,  9j(y )  II  Gm{y)} 

1  djED\di  Z— 1  djED\di  dui£.D\{d'i,dj} 

We  provide  four  heuristic  approaches  to  estimate  the  expected  utility  OU(D)  from  a 
multi-threaded  negotiation  composed  by  the  threads  D  [6]: 

•  Conservative  estimation:  The  utility  of  the  buyer  is  equal  to  the  expected  sec¬ 
ond  highest  maximum  utility.  This  approach  ignores  the  further  concession  of 
the  winning  seller  in  the  continued  single-threaded  bargaining  process  after  she 
outbids  the  other  opponents.  The  expected  utility  is  calculated  by 

OU  =  [V  yg2(y)dy 
Jo 
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where  y  is  the  upper  bound  of  the  possible  utility  that  the  negotiator  can  achieve. 
If  the  lower  bound  of  an  acceptable  price  for  a  seller  is  c,  and  the  upper  bound 
of  a  buyer’s  valuation  is  v,  then  y  —  v  —  c. 

•  Medium  estimation:  Assume  the  continued  single-threaded  bargaining  ends  at 
the  middle  point  between  the  buyer’s  and  the  winning  seller’s  reservation  price, 
if  the  buyer’s  reservation  price  is  higher  than  the  winning  seller’s4.  Then  the  ex¬ 
pected  utility  is  the  average  of  the  expected  highest  and  second  highest  maximum 
utility. 

OU  =  ( fVyg2(y)dy+  fV  yg\y)dy)/2 
Jo  Jo 

In  this  estimation  we  do  not  consider  the  probability  that  the  negotiation  may 
fail  even  if  an  agreement  is  actually  desirable  for  both  parties.  This  is  because 
in  a  negotiation  model  with  incomplete  information,  negotiators  are  not  willing 
to  reveal  their  reservation  prices  but  expect  the  concessions  of  the  other.  This 
inefficiency  is  considered  in  the  approach  of  uniform  approximation. 

•  Uniform  approximation:  Previous  research  has  established  an  optimal  bargaining 
result  between  a  buyer  and  a  seller  based  on  game  theoretic  analysis  when  both 
parties’  reservation  prices  follow  uniform  distributions  [11,  3].  Based  on  this  re¬ 
sult,  an  agreement  occurs  if  and  only  if  the  buyer’s  valuation  exceeds  the  seller’s 
cost  by  at  least  1/4,  if  both  parties’  reservation  prices  distribute  uniformly  on 
[0, 1].  In  other  words,  an  agreement  cannot  be  reached  if  the  buyer’s  valuation  is 
less  than  the  seller’s  cost  plus  1/4  of  the  maximum  difference  between  the  buyer’s 
valuation  and  the  seller’s  cost.  We  can  approximate  the  probability  distributions 
of  negotiators’  types  by  uniform  distributions  and  apply  this  result  to  calculate 
the  probability  of  reaching  an  agreement.  In  the  heuristic  we  assume  an  agree¬ 
ment  cannot  be  reached  in  the  continued  single-threaded  negotiation  between  the 
buyer  and  the  winning  seller  if  the  maximum  utility  of  the  winning  seller  is  less 
than  a  quarter  of  the  highest  possible  utility  y.  In  this  case  the  buyer  achieves 
the  second  highest  maximum  utility,  which  is  the  reservation  utility  of  the  buyer 
in  the  continued  single-threaded  negotiation.  If  an  agreement  is  reached  in  the 
single-threaded  negotiation,  it  is  reasonable  to  assume  that  it  is  at  the  middle 
point  between  both  parties’  reservation  prices.  Therefore  in  this  case  the  buyer 
achieves  the  medium  of  the  highest  and  the  second  highest  maximum  utility. 

ou  =  r  y  (,>*,+  /Vfe)A,(i  -  r  Mm 

2  Jy/4  Jo  Jy/ 4 

4If  the  buyer’s  reservation  price  is  lower  than  the  seller’s,  there  is  no  “zone  of  agreement”  and  the 
negotiation  will  fail. 
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•  Learning:  Learn  the  probability  of  reaching  an  agreement  and  the  distribution 
of  agreements  based  on  the  previous  negotiations  [14].  The  result  of  learning  is 
represented  by  x(Rb,Rs),  the  expected  agreement  on  the  price  from  the  negotia¬ 
tion  when  the  buyer’s  and  seller’s  reservation  prices  are  Rb  and  Rs  respectively. 
Given  the  probability  distribution  of  the  opponent’s  reservation  price,  a  negotia¬ 
tor  can  calculate  the  expected  utility  of  the  negotiation  based  on  the  result  of 
learning.  If  the  seller  st  in  the  thread  d*  is  the  winning  seller,  then  the  probability 
distribution  of  her  reservation  price  is  F(c)  [Ad:ien\d,{^  —  F(vj  ~  vi  +  c))>  where 
the  product  is  the  probability  that  no  other  thread  dj  has  the  maximum  utility 
Vj  —  Cj  greater  than  the  maximum  utility  u,  —  c,  in  thread  dj.  Then  the  expected 
utility  from  a  multi-threaded  negotiation  can  be  calculated  by 

OU  =  /  ( l’d  -  x(vi,  c))  [Q  (1  -  F(vj  -  Vi  +  c))dF(c) 

di&D  —  dj&D\di 


If  negotiators  use  the  time-dependent  strategy  and  the  parameter  (3  is  chosen 
randomly  with  the  mean  equal  to  1,  then  we  expect  negotiators  to  concede  con¬ 
stantly  on  aggregation.  Then  the  result  of  learning  is  expected  to  be  close  to  the 
result  of  negotiation  when  j3  —  1  for  both  negotiators.  Let  the  reservation  prices 
of  the  buyer  and  the  seller  be  v  and  c  respectively.  Then 


x(v,  c ) 


1 +V—C 


if  v  >  c 
otherwise 


(6) 


assuming  the  upper  bound  of  an  offer  is  1  and  the  lower  bound  is  0.  This  formula 
can  be  derived  from  Figure  2  that  shows  the  offer  curves  of  the  negotiators  with 
[3  —  1  for  both  parties: 


2.3  Dynamic  multi-threaded  negotiations 

In  the  Navy  detailing  process  the  application  period  for  a  position,  or  search  period 
for  filling  a  job,  lasts  for  some  months.  During  that  period  potential  partners  are  dis¬ 
covered  sequentially  and  new  negotiations  are  launched  dynamically.  For  an  ongoing 
negotiation  thread  the  outside  options  not  only  include  the  other  simultaneous  nego¬ 
tiation  threads,  but  also  the  threads  that  may  be  launched  in  the  future.  Considering 
the  outside  options  in  the  future,  a  negotiator  must  decide  how  much  to  offer  in  the 
current  negotiation,  and  when  to  stop  searching  for  future  opportunities  and  accept 
an  offer  from  the  current  negotiation.  If  a  negotiator  knows  the  number  of  outside 
options  that  will  come,  and  the  value  of  the  opponent  in  each  outside  option,  then  the 
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Figure  2:  Offer  curves  with  (3  —  1 

negotiator  can  apply  the  synchronized  multi-threaded  negotiation  model  to  calculate 
the  appropriate  reservation  price  in  each  thread.  But  in  the  Navy  detailing  process, 
neither  a  Command  nor  a  Sailor  is  sure  about  the  arrival  of,  and  the  opponents’  values 
in,  future  outside  options.  The  reservation  utility  of  a  thread  is  the  expected  utility 
of  a  multi-threaded  negotiation  -  including  other  simultaneous  threads  and  threads 
launched  in  the  future  -  with  a  stochastic  thread  number  and  uncertain  opponents. 

Following  the  usual  way  of  modelling  uncertain  arrival,  we  assume  the  arrival  of  outside 

options  follows  a  Poisson  process  [9,  10,  13].  In  each  period  t,  t  =  0, _ ,  T  —  1,  there 

is  probability  p  that  the  buyer  finds  a  matching  alternative  and  launches  a  negotiation 
thread.  The  granule  of  each  period  is  small  enough  so  that  the  probability  that  there 
are  more  than  one  arrival  in  one  period  is  zero.  Again  the  value  v  of  a  seller  follows 
the  probability  distribution  $(?/)  =  Pr{y  <  y),  where  $(?/)  is  the  probability  that 
a  seller’s  value  is  no  greater  than  y.  The  reservation  price  Rs  of  a  seller  follows  the 
prior  belief  F(x)  =  Pr(Rs  <  x).  A  Command  can  evaluate  a  Sailor  by  checking  the 
Sailor’s  background.  A  Sailor  also  knows  how  much  he  prefers  a  job  by  acquiring  the 
job  information  about  location,  responsibility,  etc.  But  how  much  a  Command  values 
a  Sailor  or  a  Sailor  values  a  Command  is  unknown  to  the  Sailor  or  the  Command 
respectively.  Therefore  a  negotiator  knows  the  value  of  an  opponent  when  the  opponent 
is  identified,  but  not  the  reservation  price  of  the  opponent. 

The  state  St  of  the  system  is  defined  as  the  number  of  threads  rit  and  the  value  of  each 
opponent  seller  vj,  st  =  {nt,  {^rf}2=i } -  The  evolution  of  the  system  follows  the  rule 

/  int  +  1;  {ud}]] Li  U  v}  if  an  opponent  with  value  v  arrives  at  period  t 
St+1  [  St  if  no  arrival  at  period  t 
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Let  OUt(st)  be  the  utility  that  the  negotiator  expects  from  the  dynamic  multi-threaded 
negotiation  when  she  sees  the  system  state  ,st  at  period  t.  Following  Section  2.2  we  can 
calculate  OU({n,{vd}^=1}),  the  expected  utility  from  a  synchronized  multi-threaded 
negotiation  with  n  threads  and  the  opponent  in  thread  d  valued  v^,  d  —  1, ,  n.  The 
transition  of  the  expected  utility  follows  the  rule 

OUt(st)  =  (1  -p)OUt+1(st)  +pEv[OUt+1({nt  +  1,  {^Uv})],  (7) 

OUt-i(st-i)  =  OU(st~  i). 

If  the  probability  of  arrival  at  each  period  is  p,  then  the  number  of  arrivals  r)(m,p) 
during  an  interval  with  length  r  follows  a  Poisson  distribution,  PpT(n)  =  Pr(r](T,p)  = 
n )  =  e~pT  ^  .  Equivalently  we  can  write  the  transition  of  the  expected  utility  as 

OUt(st)  =  En[E.  ,».«  \OU({nt  +  t,  {„„}£,  U  {MS+iHI  (8) 

L  “J  d=n^-\- 1 

where  rj  following  a  Poisson  distribution  PpT_t(-),  and  independently  follows  the 
identical  distribution  <£>(•),  d  —  nt  +  1, . . . ,  nt  +  r\. 

To  set  the  reservation  price  of  a  thread,  the  negotiator  only  needs  to  calculate  the 
expected  utility  of  the  multi-threaded  negotiation  which  does  not  include  that  thread, 
based  on  the  period  and  real-time  state.  Because  the  state  of  a  dynamic  multi-threaded 
negotiation  changes  from  period  to  period,  the  reservation  price  of  a  thread  may  also 
change  with  time. 

The  expected  utility  of  a  dynamic  multi-threaded  negotiation  process  at  each  period 
with  each  state  can  be  calculated  backward  from  the  last  period  following  Equation  7 
or  forward  following  Equation  8.  The  computation  will  be  very  expensive  to  calculate 
the  expectation  of  the  expected  utility  on  the  opponents’  values.  If  there  are  at  most 
N  threads  and  for  each  opponent  there  are  M  possible  values,  then  the  number  of 
possible  states  will  be  NM .  The  computation  is  intractable  with  large  M.  To  simplify 
the  computation  we  can  approximate  the  result  by  having  the  opponent  value  instances 
replaced  by  the  expected  value  v,  i.e. , 

OUt(st )  =  (1  -p)OUt+1(st)  +pOUt+1({nt  +  l,{vd}1£=1  Uh})]  (9) 

which  is  equivalent  to 

OU,(st)  =  Ev[OU({nt  +  t,  MSU  U  {8}S+i}]]  (10) 

The  compromise  due  to  this  simplification  is  not  significant  if  the  expected  utility  of  a 
synchronized  thread  is  or  can  be  approximated  by  a  linear  function  of  the  opponents’ 
values. 
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3  Experiments 


In  Section  2  we  have  presented  two  models  of  the  outside  options,  the  synchronized  and 
dynamic  multi-threaded  negotiation  models,  and  four  heuristic  approaches,  the  con¬ 
servative  estimation,  the  medium  estimation,  the  uniform  approximation  and  learning 
approach,  to  estimate  the  expected  utility  in  a  multi-threaded  negotiation.  By  com¬ 
bining  different  outside  option  models  and  estimation  approaches,  we  can  have  eight 
decision  models  for  bilateral  negotiations  in  the  Navy  detailing  process.  In  this  section 
we  provide  experiments  to  illustrate  the  different  models  in  the  solution  framework  and 
the  performance  results  based  on  the  different  models.  We  used  the  time-dependent 
strategy  as  the  strategy  in  a  single-threaded  negotiation  because  it  is  simple  to  com¬ 
pute.  In  the  solution  framework  that  we  have  presented,  the  reservation  utility  is  an 
important  system  variable  that  decides  the  reservation  price,  which  impacts  the  of¬ 
fer  curve  based  on  a  specific  negotiation  strategy.  In  Section  3.1  we  show  how  the 
reservation  utility  of  a  negotiation  thread  evolves  with  time  and  the  change  of  outside 
options  in  the  synchronized  and  dynamic  multi-threaded  negotiation  models.  We  then 
show  the  impact  of  outside  options  on  the  negotiation  strategy  by  showing  the  offer 
curves  adjusted  by  the  reservation  prices,  compared  with  the  original  basic  offer  curve 
without  considering  outside  options.  In  Section  3.2  we  compare  the  average  utility 
of  a  negotiator  when  she  (1)  does  not  consider  outside  options,  (2)  when  she  only 
considers  concurrent  outside  options,  i.e.,  the  synchronized  multi-threaded  negotiation 
model,  and  (3)  when  she  considers  both  concurrent  outside  options  and  future  arrivals, 
i.e.,  the  dynamic  multi-threaded  negotiation  model.  The  performance  results  based  on 
different  utility  estimation  approaches  are  also  compared  and  discussed. 

3.1  Reservation  utilities  and  offer  curves 

We  illustrate  the  impact  of  outside  options  on  the  negotiation  strategy  by  a  specific 
example.  In  this  example  the  negotiator  is  a  buyer.  The  negotiation  deadline  T  =  20. 
The  buyer  believes  the  reservation  price  of  a  seller  follows  a  uniform  distribution  on 
the  interval  [0,1].  The  value  of  (the  item  of)  a  seller  to  the  buyer  is  also  uniformly 
distributed  on  [0,1].  In  each  period  with  probability  p  >  0  a  new  seller  arrives  and 
a  negotiation  thread  is  created.  The  shape  of  the  offer  curve  defined  by  Equation  3 
and  1  (Section  2.1)is  determined  by  the  parameter  (3 .  We  ran  multiple  experiments 
with  p  =  0.05,0.10,0.15,0.20,0.25  and  random  (3.  The  resulting  curves  followed  the 
same  pattern  for  all  instances.  We  show  the  resulting  curves  based  on  one  instance 
with  p  =  0.2  and  f3  =  1.262727.  The  arrivals  of  outside  options  in  the  instance  are 
illustrated  in  Figure  3.  The  figure  on  the  left  shows  the  time  of  arrivals  and  the  value 
of  each  arrival,  the  figure  on  the  right  shows  the  number  of  threads  in  each  period. 
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Number  of  total  arrivals 


Time  and  values  of  arrivals 


Figure  3:  Arrivals  and  number  of  threads 


To  illustrate  the  evolution  of  the  reservation  utility  of  a  thread,  we  collect  the  reserva¬ 
tion  utility  of  the  first  thread  along  time  calculated  with  different  estimation  approaches 
and  models  of  the  outside  options.  Figure  4  shows  the  reservation  utilities  grouped  by 
the  estimation  approach  and  Figure  5  compares  the  reservation  utilities  calculated 
based  on  different  estimation  approaches  but  on  the  same  outside  option  model.  The 
expected  utility  of  a  dynamic  multi-threaded  negotiation  process  was  calculated  with 
the  approximation  formula,  Equation  10  (Section  2.3). 

We  can  see  from  Figure  4  that  the  reservation  utility  based  on  the  synchronized  model 
without  expecting  arrivals  in  the  future  is  less  than  the  reservation  utility  calculated 
based  on  the  dynamic  model  where  the  probable  future  arrivals  are  also  taken  into 
consideration5.  This  is  as  expected  because  in  the  dynamic  model  a  negotiator  does 
not  only  see  the  current  existing  outside  options,  but  also  foresees  the  outside  options 
that  may  come  in  the  future.  Although  the  number,  the  arrival  times  and  opponents’ 

5There  is  a  region  in  which  the  reservation  utility  based  on  the  synchronized  model  is  slightly 
higher  than  the  one  based  on  the  dynamic  model.  This  is  because  of  the  learning  model  that  is  used. 
Consider  the  situation  where  there  is  only  one  thread  d\ .  The  expected  utility  based  on  the  learning 
model  is  V\  ■  Now  assume  a  new  negotiation  opponent  comes,  and  the  value  of  the  opponent 

is  v2  and  the  reservation  price  is  C2-  The  estimation  model  suggests  that  the  expected  utility  from 
the  two-threaded  negotiation  is  v2 1^~^.2C2 Pr{v2  —  c2  >  i>i  —  Ci)  +  V\  Pr(vi  —  ci  >  v2  —  c2). 

When  V\  is  much  greater  than  v2,  v2  -^A-co  maY  be  less  than  v\  1^_1v~^_1Ci  even  if  v2  —  c2  is  greater 
than  v\  —  C\.  Therefore  more  threads  do  not  necessarily  mean  higher  expected  utility.  Arrivals  with 
very  low  values  may  actually  reduce  the  expected  utility  calculated  by  the  learning  approach.  But 
generally  we  can  say  that  the  expected  utility  increases  with  the  number  of  threads  and  hence  the 
expected  utility  based  on  the  dynamic  model  is  higher  than  the  one  based  on  the  synchronized  model. 


16 


Reservation  utility  with  conservative  estimation  Reservation  utility  with  medium  estimation 


Figure  4:  Reservation  utilities  grouped  by  estimation  approaches 


values  of  the  future  outside  options  are  uncertain,  they  are  still  valuable  and  provide 
opportunities  of  reaching  an  agreement  outside  the  current  negotiation  threads.  There¬ 
fore  the  expectation  on  the  possible  future  outside  options  raises  the  lowest  utility  that 
a  negotiator  can  accept  in  a  current  negotiation  thread. 

Figure  5  shows  that  the  approach  of  medium  estimation  suggests  a  higher  reservation 
utility  than  the  other  approaches  and  gives  the  most  optimistic  estimation  on  the  util¬ 
ity  from  outside  options.  The  medium  estimation  gives  a  more  optimistic  estimation 
than  the  conservative  estimation  because  in  the  latter  the  concession  of  the  winning 
opponent  in  the  continued  single-threaded  negotiation  is  ignored.  The  expected  utility 
based  on  the  medium  estimation  is  higher  than  on  the  uniform  approximation  estima¬ 
tion  because  the  inefficiency  of  a  negotiation  with  two-sided  incomplete  information  is 
considered  in  the  latter  but  not  in  the  former.  The  medium  estimation  also  suggests  a 
higher  expected  utility  than  the  learning  approach  because  in  the  latter  the  negotiation 
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Reservation  utilities  with  the  synchronized  model  Reservation  utilities  with  the  dynamic  model 


Figure  5:  Reservation  utilities  grouped  by  outside  option  models 


outcome  in  the  continued  single-threaded  negotiation  is  not  compared  with  the  second 
highest  maximum  utility,  while  in  the  former  it  is  guaranteed  that  the  negotiation  out¬ 
come  in  the  continued  single-threaded  negotiation  is  not  worse  than  the  second  highest 
maximum  utility. 

No  matter  which  estimation  approach  is  used,  Figure  5  shows  that  the  reservation 
utility  based  on  the  synchronized  model  (Section  2.2)  monotonically  increases  with  time 
because  the  number  of  threads  increases  with  time.  But  it  is  interesting  to  note  that 
the  reservation  utility  based  on  the  dynamic  model  (Section  2.3)  is  not  a  monotonic 
function  of  time.  This  is  because  there  are  two  forces  that  drive  the  change  of  the 
reservation  utility:  time  and  concurrent  threads.  When  the  negotiator  approaches  the 
deadline,  the  possibility  to  have  new  arrivals  decreases  and  it  drives  the  reservation 
utility  down.  On  the  other  hand  the  reservation  utility  would  increase  with  the  arrival 
of  a  new  negotiation  opponent,  especially  when  the  value  of  the  new  opponent  is 
high.  From  Figure  5  we  can  see  that  the  reservation  utility  has  different  sensitivity 
to  the  change  of  time  and  new  arrivals  based  on  different  estimation  approaches.  The 
reservation  utility  based  on  the  learning  approach  does  not  change  as  much  as  the 
ones  based  on  other  approaches  as  time  or  outside  options  vary.  No  matter  which 
estimation  approach  or  outside  option  model  is  used,  the  resulting  reservation  utility 
with  consideration  of  future  outside  options  is  higher  than  without  considering  the 
future  outside  options. 

Based  on  the  reservation  utility  OUd  of  thread  d  we  can  calculate  the  reservation  price 
of  the  buyer,  Rd  =  vd  —  OUd,  where  vd  is  the  value  of  the  seller  in  thread  d.  We 
compare  the  offer  curves  based  on  different  estimation  approaches  and  outside  option 
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models  in  Figure  6.  The  model  noted  by  “Single”  is  the  model  without  considering 
outside  options.  In  that  model  the  reservation  price  Rd  is  equal  to  the  opponent’s 
value  Vd  and  the  offer  xt  in  period  t  is  calculated  by  xt  =  (t/T)l^Vd  following  Equation 
1  (We  assume  the  lower  bound  mini >  °f  a  valid  offer  is  zero.).  Without  considering 
outside  options,  the  offer  increases  with  time  as  the  buyer  constantly  concedes  (with 
changing  pace).  But  with  a  synchronized  or  dynamic  model  the  buyer  may  proceed, 
i.e.,  decrease  the  offered  price  from  the  previous  one,  when  a  valuable  new  opponent 
arrives.  The  pace  of  concession  is  also  different  with  different  outside  options  models. 
When  a  valuable  seller  arrives,  the  buyer  may  proceed  by  asking  for  a  lower  price 
than  the  previous  offer  because  the  buyer  gets  more  optimistic  about  the  expected 
utility  that  she  can  get  from  the  outside  options.  In  the  dynamic  model  the  speed 
of  concession  at  some  time  may  be  higher  than  without  considering  outside  options 
in  the  single  model,  because  of  the  impact  of  both  the  increasing  time  pressure  on 
reaching  an  agreement  and  the  decreasing  hope  on  the  availability  of  future  outside 
options.  The  offers  without  considering  outside  options  are  higher  than  the  offers  with 
considering  only  concurrent  negotiation  threads,  which  are  again  higher  than  the  offers 
with  additional  considerations  of  outside  options  that  may  come  in  the  future.  This  is 
consistent  with  the  observation  that  the  reservation  utility  based  on  the  synchronized 
model  is  less  than  the  one  based  on  the  dynamic  model. 

3.2  Performance  results 

In  this  section  we  examine  and  compare  the  average  utilities  that  a  buyer  obtains  with 
three  different  outside  option  models  and  four  different  estimation  approaches.  The 
three  outside  option  models  include:  (1)  the  “Single”  model  in  which  no  outside  op¬ 
tions  are  considered,  (2)  the  “Synchronized”  model  in  which  only  concurrently  existing 
negotiation  threads  are  considered  as  outside  options,  and  (3)  the  “Dynamic”  model  in 
which  the  outside  options  also  include  the  possible  uncertain  future  arrivals.  The  four 
estimation  approaches  include  the  conservative  estimation,  the  medium  estimation,  the 
uniform  approximation  and  the  learning  approach.  In  the  experiments  the  negotiation 
deadline  T  =  20.  The  buyer  believes  the  reservation  price  of  a  seller  follows  a  uni¬ 
form  distribution  on  the  interval  [0,1].  The  value  of  a  seller’s  item  is  also  uniformly 
distributed  on  [0, 1].  The  probability  that  a  new  seller  arrives  in  a  period  is  p,  and  p 
takes  the  values  {0.05,0.10,0.15,0.20,0.25}.  The  parameter  f3  in  the  time-dependent 
strategy  of  a  negotiator  is  chosen  randomly  so  that  with  even  probability  a  negotiator 
in  a  thread  is  a  conceder  (/3  >  1)  or  a  boulware  (/?  <  1).  If  a  negotiator  is  a  conceder, 
(5~l  follows  a  uniform  distribution  on  [0, 1].  If  a  negotiator  is  a  boulware,  /3  is  a  random 
variable  with  a  uniform  distribution  on  [0,1].  For  each  arrival  probability,  we  repeat 
the  experiment  50  times  and  the  average  utility  of  the  buyer  is  calculated. 
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The  offer  curve  with  conservative  estimation 


The  offer  curve  with  medium  estimation 


The  offer  curve  with  estimation  by  uniform  approximation  The  offer  curve  with  estimation  by  learning 


Figure  6:  The  offer  curves 


Figure  7  is  composed  of  four  subplots.  Each  subplot  shows  the  average  utility  as  a 
function  of  the  arrival  probability  based  on  one  estimation  approach,  and  with  differ¬ 
ent  outside  option  models.  The  figure  implies  that  for  all  estimation  approaches  and 
outside  option  models,  the  average  utility  increases  with  the  arrival  probability.  This 
is  intuitive  and  should  be  true  for  a  reasonable  negotiation  strategy.  A  higher  arrival 
probability  implies  more  options  on  expectation  and  should  result  in  better  outcome 
for  the  negotiator.  Figure  7  also  shows  that  the  average  utility  based  on  the  dynamic 
model  is  higher  than  the  one  based  on  the  synchronized  model,  which  again  brings 
higher  average  utility  than  the  single-threaded  model  in  which  no  outside  option  is 
considered.  This  verifies  the  effectiveness  of  the  outside  option  models  that  we  have 
proposed. 

We  can  also  group  the  average  utilities  by  the  outside  option  model  and  compare  the 
performance  of  the  estimation  approaches.  The  information  is  shown  in  Figure  8.  The 
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Average  utility  with  conservative  estimation 


Average  utility  with  medium  estimation 


Average  utility  with  estimation  by  uniform  approximation  Average  utility  with  estimation  by  learning 


Figure  7:  The  average  utilities  grouped  by  estimation  approaches 


figure  shows  that  there  is  no  estimation  approach  that  dominates  the  others.  This  is 
because  the  performance  of  an  approach  depends  on  the  negotiators’  offer  curves.  If 
both  negotiators  tend  to  concede  quickly  (/3  is  very  large),  an  optimistic  estimation 
approach  such  as  the  medium  approach  may  be  better.  On  the  other  hand  if  both 
negotiators  tend  to  hold  on  their  positions  (/ 3  is  very  small),  the  conservative  estimation 
approach  may  be  better. 


4  Discussion  and  conclusion 

In  this  report  we  provide  an  integrative  solution  for  the  negotiation  decision  problem 
in  the  Navy  detailing  system  when  negotiators  face  uncertain  and  dynamic  outside 
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Average  utility  with  the  sychronized  model 


Average  utility  with  the  dynamic  model 


0.15 
Arrival  probability 


Figure  8:  The  average  utility  grouped  by  outside  option  models 


options.  The  outside  options  influence  the  negotiation  strategies  via  the  impact  on  the 
reservation  prices.  The  solution  is  composed  of  three  modular  models:  single-threaded 
negotiations,  synchronized  multi-threaded  negotiations,  and  dynamic  multi-threaded 
negotiations.  The  single-threaded  negotiation  model  provides  the  negotiation  strategy 
given  the  reservation  price.  The  other  two  models  calculate  the  reservation  price  based 
on  the  model  of  the  outside  options.  The  model  of  synchronized  multi-threaded  negoti¬ 
ations  considers  the  presence  of  concurrently  available  outside  options  and  provides  an 
approach  to  estimate  the  outcome  when  the  threads  are  known.  The  model  of  dynamic 
multi-threaded  negotiations  expands  the  last  model  by  considering  the  uncertain  out¬ 
side  options  that  may  come  dynamically  in  the  future.  The  specific  solution  for  each 
module  is  presented,  and  experimental  analysis  is  provided.  The  results  show  that 
the  utility  of  a  negotiator  improves  significantly  when  she  considers  outside  options 
than  when  she  does  not  consider  them,  and  when  she  considers  the  dynamic  arrival 
of  outside  options  than  when  she  only  considers  the  concurrently  existing  negotiation 
threads. 

We  would  like  to  make  the  following  remarks  to  avoid  possible  confusions  in  under¬ 
standing  the  model: 

-  We  take  an  artificial  intelligence  approach  instead  of  a  game  theoretic  or  eco¬ 
nomics  approach  in  this  study  because  the  complexity  of  the  situation  does  not 
allow  rigorous  mathematical  analysis,  which  is  usually  preferred  by  economists. 
The  approaches  in  the  AI  field  are  different  from  the  models  in  economics  in  that 
AI  approaches  aim  to  provide  an  effective  heuristic  solution  to  general,  realistic 
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and  complicated  situations,  since  in  complicated  models  there  is  lack  of  rigorous 
mathematical  analysis. 

-  We  are  not  using  an  auction  mechanism6.  In  a  multi-threaded  negotiation  pro¬ 
cess,  each  negotiation  thread  is  an  outside  option  of  other  threads.  But  in  an 
auction  which  includes  all  candidates  as  bidders,  there  is  no  outside  option  for 
the  auctioneer. 

-  The  ’’reservation  price”  should  not  be  confused  with  the  ’’intrinsic  value”  of  an 
item.  The  reservation  price  from  a  buyer’s  point  of  view  is  the  highest  price  that 
is  acceptable  in  one  negotiation.  The  highest  acceptable  price  is  no  greater  than 
the  intrinsic  value.  When  there  are  outside  options,  the  highest  acceptable  price 
may  be  less  than  the  intrinsic  value. 

-  Outside  options  change  the  reservation  price  in  a  negotiation.  To  give  a  very 
simple  example,  if  a  buyer  knows  that  she  could  buy  an  item  from  a  seller  for 
$20,  she  will  not  buy  it  from  another  seller  for  more  than  $20,  although  she  values 
the  item  at  $25.  In  the  Navy  situation,  a  buyer  is  uncertain  about  the  availability 
and  quality  of  outside  options,  neither  does  she  know  the  exact  agreement  she 
could  reach  in  other  outside  negotiations.  She  could  only  set  an  appropriate 
reservation  price  by  estimating  the  utility  she  could  achieve  from  outside  options 
with  reasonable  heuristic  approaches,  which  we  have  provided  in  our  deliverable. 

In  this  negotiation  solution  we  have  focused  on  the  negotiation  strategy  when  the 
negotiator  faces  uncertain  outside  options.  We  did  not  explicitly  model  the  behavior 
of  the  negotiation  opponents  when  they  also  have  outside  options.  The  outside  options 
of  an  opponent  are  unknown  to  the  negotiator  and  influence  the  reservation  price  of 
the  opponent.  Since  the  reservation  price  is  private  information,  the  outside  options 
of  an  opponent  can  be  taken  into  consideration  if  the  prior  belief  on  the  opponent’s 
reservation  price  also  includes  the  probabilistic  information  on  her  outside  options. 

In  this  report  we  propose  heuristic  solutions  for  the  negotiation  decision  problem  in  the 
Navy  detailing  process.  The  complexity  of  interactions  in  an  alternating-offers  bilateral 
negotiation  with  two-sided  asymmetric  information  does  not  allow  the  mathematical 
analysis  of  optimal  strategies  with  general  settings  such  as  continuous  type  space  and 
general  probability  distribution  of  the  prior  belief,  even  without  outside  options.  In 
this  work  we  pursue  the  practical  effectiveness  of  the  solution.  We  have  proposed  and 
applied  negotiation  strategies  that  have  been  developed  by  us  and  others  in  the  AI  field, 
and  we  have  provided  reasonable  heuristic  approaches  to  set  appropriate  reservation 

6Even  with  auctions,  the  reservation  price  is  not  necessarily  equal  to  the  true  value  [Auction  Theory, 
Vijay  Krishna,  Academic  Press,  2002]. 
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prices  considering  outside  options.  Existing  results  from  economics  on  the  optimal 
solutions  of  simpler  models,  such  as  auctions  and  bilateral  negotiations  with  unform 
distributions  of  the  prior  beliefs,  are  used  to  design  reasonable  heuristics  to  solve  the 
complicated  problem  in  our  model.  Because  of  the  heuristic  approach  that  we  take 
in  this  work,  extensive  simulations  are  needed  to  provide  rigorous  evaluation  on  the 
performance  of  different  models  and  approaches  in  different  environments.  We  do  not 
claim  that  the  heuristics  we  provide  in  this  report  are  complete.  Rather  they  reflect 
solutions  that  have  been  proven  useful  or  plausible.  Other  negotiation  strategies  and 
approaches  to  estimate  the  utility  from  a  multi-threaded  negotiation  can  be  plugged 
in  the  solution  framework,  depending  on  the  assumptions  and  requirements  of  the 
underlying  application.  These  different  models  can  construct  a  library  of  decision 
functions  to  support  the  decision  of  negotiation  agents  in  different  environments. 

Bilateral  negotiations  are  a  useful  mechanism  to  realize  distributed  matching  between 
Sailors  and  Commands  that  do  not  get  matched  through  the  mass-matching  market. 
How  these  two  mechanisms  interact  with  each  other  depends  on  Navy  policy.  If  a  Com¬ 
mand  or  a  Sailor  has  to  accept  the  matching  result  from  the  mass- matching  market, 
then  the  bilateral  negotiations  can  be  regarded  as  outside  options  of  the  mass-matching 
market.  How  much  to  bid  in  the  mass-matching  market  is  impacted  by  the  reservation 
utility  that  a  Command  or  a  Sailor  expects  to  obtain  via  bilateral  negotiations.  Other¬ 
wise  if  a  Command  or  a  Sailor  can  reject  the  results  from  the  mass-matching  market, 
then  the  bidding  decision  is  not  entangled  with  the  bilateral  negotiation  process.  But 
the  decision  to  accept  or  reject  the  outcome  of  the  mass-matching  mechanism  still 
depends  on  the  expected  utility  from  the  bilateral  negotiations. 
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